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Objective: 

To  investigate  ways  to  increase  the  affordability  of  human  behavioral  modeling  through 
development  of  better  software  tools  to  support  this  task  and  the  gathering  of  objective  evidence 
that  the  tools  do  make  modeling  more  affordable. 

Approach: 

We  proposed  to  develop  a  suite  of  Cognitive  Tutor  Authoring  Tools  (CTAT)  to  make 
modeling  both  easier  and  faster  for  experienced  modelers  and  possible  for  potential  modelers 
who  are  not  experts  in  cognitive  psychology  or  artificial  intelligence  programming.  Our  concrete 
goal  was  to  reduce  modeling  time  by  a  factor  of  three  and  to  experimentally  demonstrate  this 
reduction.  We  also  proposed  to  push  the  state  of  the  art  in  “programming  by  demonstration”  by 
having  our  tools  induce  cognitive  rules  from  demonstrated  solutions. 

The  approach  we  used  was  to  build  a  suite  of  tools,  called  the  Cognitive  Tutor  Authoring 
Tools  (CTAT)),  which  allow  non-programmers  to  “demonstrate”  the  behavior  they  want  the 
system  to  model,  A  central  tool  called  the  Behavior  Recorder  records  the  demonstrated  problem 
steps  in  a  “Behavior  Graph,”  which  may  capture  multiple  paths  through  a  problem  scenario,  as 
well  as  incorrect  behavior  that  needs  to  be  modeled.  Behavior  Graphs  are  used  in  two  ways. 
First,  they  are  used  to  build  what  we  call  “example-tracing  tutors,”  which  are  less  general  than 
cognitive  rule-based  tutors  but  much  faster  to  develop.  The  second  use  of  these  demonstrations 
is  as  a  guide  to  help  behavioral  rule-based  modelers  do  their  task.  The  cognitive  modeler  uses 
the  Behavior  Graphs  as  test  cases  to  test  his/her  rules.  The  CTAT  tools  are  shown  below  and 
described  in  greater  detail  in  Koedinger,  Aleven,  Heffeman,  McLaren,  &  Hockenberry,  2004; 
Aleven,  Sewall,  McLaren,  &  Koedinger,  in  press;  Aleven,  McLaren,  Sewall,  &  Koedinger,  in 
press. 


We  used  these  tools  to  build  our  own  tutors  as  well  as  hosting  weeklong  summer  workshops 
in  2003,  2004,  and  2005  in  which  we  freely  distributed  the  tools  to  researchers  and  trained  them 
in  their  use.  Over  100  researchers  have  attended  these  training  and  dissemination  workshops  and 
several  other  academic  groups  are  now  using  the  tools.  We  have  also  made  our  tools  freely 


available  from  a  website  (http://ctat.pact.cs.cmu.edu/).  We  estimate  that  there  are  currently 
between  100  and  200  users  of  CTAT. 

Accomplishments: 

We  were  able  to  show  that  our  tools  can  be  used  to  build  many  different  rule-based  cognitive 
models,  such  as  a  Warrior  Simulator  at  Fort  Benning  (Livak  &  Heffeman,  2004),  a  Logic  Tutor 
for  internal  use  at  CMU,  and  a  Genetics  tutor. 

In  preliminary  controlled  experiments  involving  basic  Cognitive  Tutor  development  tasks, 
we  found  efficiency  gains  due  to  CTAT  of  1.4  to  2  times  faster  (Aleven,  McLaren,  Sewall  & 
Koedinger,  2006).  We  also  demonstrated  across  4  different  behavioral  modeling  projects  that 
our  tools  created  example-tracing  tutors  that  drastically  reduced  modeling  costs  (Koedinger, 
Aleven,  Heffeman,  McLaren,  &  Hockenberry,  2004,  Heffeman,  Turner,  Lourenco,  Macasek, 
Nuzzo-Jones,  Koedinger,  2006).  Not  only  did  we  reduce  the  time  dramatically  (averaging  a 
reduction  of  over  a  factor  5),  we  also  reduced  the  experience  level  modelers  needed. 

Finally,  we  pushed  the  state  of  the  art  in  data  mining  and  machine  learning  support  to  help 
rule-writers  (McLaren  et  al  2005;  Harrer  et  al,  2005;  McLaren  et  al,  2004b;  McLaren  et  al, 

2004a;  Jarvis,  Nuzzo-Jones,  &  Heffeman,  2004;  Matsuda,  Cohen,  and  Koedinger,  2005a; 

2005b;  2005c). 

Conclusions 

We  have  made  excellent  progress  toward  creating  and  demonstrating  affordable  behavior 
modeling  through  development  and  evaluation  research  on  the  Cognitive  Tutor  Authoring  Tools. 
The  central  approach  has  been  to  use  human-computer  interaction,  data  mining,  and  machine 
learning  techniques  to  create  software  components,  smart  interfaces  and  underlying  intelligence, 
that  speed  the  process  of  behavior  modeling.  Critical  to  our  success  has  been  creating  a  large 
user  community  that  provides  both  realistic  user  tests  and  vote-with-your-feet  evidence  that 
CTAT  is  effectively  reducing  behavior-modeling  costs.  The  200  users  of  CTAT  represent  the 
largest  number  of  users  of  any  technology  in  the  AF1BM  program.  We  have  also  provided 
reasonable  evidence  of  affordability  improvements. 

Significance 

Cognitive  models  are  used  successfully  throughout  the  Armed  Forces  as  well  as  in 
educational  and  training  software.  However,  cognitive  models  have  traditionally  been  very 
expensive  to  build.  This  small  project  has  produced  valuable  knowledge  about  methods  that  can 
be  used  to  make  modeling  more  cost  effective.  This  project  has  also  created  a  specific  set  of 
tools  (CTAT)  that  embodies  some  of  this  knowledge,  and  that  set  of  tools  has  led  to  the  cost- 
effective  creation  of  many  new  cognitive  models.  These  models  in  turn  lead  to  more  knowledge 
as  exhibited  by  the  many  peer-reviewed  papers  produced  under  this  grant  or  related  to  this  grant. 
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